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A large proportion of the cost of developing 
information processing systems is devcted to the 
testing phase of systen development. At present the 
resuits of this large expenditure have been inadequate 
to suit the high reliability demanded of today's 


computer systems. 


This research develops a framework of a system 
test methodology tnat can be used to provide a 
rational approach to she design of a system. The 
methodology uses a model that provides a modularized 
functional representation of a processing system. The 
nodel is used to investigate a broad class of system 
test problens. A Simulation of the model was 
constructed. The use of simulation during system 


testing was discussed. 
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DO TIVATICN FOR SYSTEM ZST METHODOLOGY 


Software is the major expense in computer systems today. 

As an example, the Air Force allocated between one billion 
ars and one and a halr billion dollars in 1972 for 
software development. This was about three times the annual 
expenditure cn computer hardware and accounted for four to 
five percent of the Air Force budget for the year. Boehm 
NEAN, 15 3 indicates ina these high figures are 
representative of the industry as a whole. He predicts that 
by 1985 software expenditures in the Air Force will account 
for ninety percent of tne total ADP system costs. Of this 
enormous amount OF zoney spent on software, a 
disproportionately large share was spent on testing and the 
rend is not one of improveaent. Boehm states that "during 

the 1970s the Air Force can expect to spend almost half of 
its software budget for military space operations on the 
checkout and test phases of computer program implementation: 
two to three times as nuch as it will pay for having the 
puogram coded." With such an effort going into testing 
software, it should be reiatively error free but this has 
not been the case historically. The Apollo Manned 
Spacefiight Program had ore of the most tested systems in 
the world, yet major software failures occured in Apollos 8, 
and iur The failure on Apollo 11 occurred in the 
extremely critical phase of lunar landing. Things are no 


better on the civilian side, each new release of 05/360 has 





around 1000 new software errors, It is not necessary to 
look at such large complicated systems to discover that 
ERcsent o testing. 1s inadequate. The person who has not had 
an encounter with a computer program error such as an 
incorrect billing is an unusuai person in today's society. 
Since testing consumes such a large proportion of the 
resources allocated to system development and has produced 
such poor results, it is time to develop a new approach to 


BN stem test. 


B. TESTING PROBLEMS 


meo qe — — AQ — Oe ee om nn 


Many of the terms used in the area of testing are 
subject to a vide variety of interpretations. The word 
"testing" has been misused and many non-testing activities 
have been associated with the word. Testing may be defined 
to be the process of determining if a system meets the 
Stated functional specifications. Quite often debugging is 
mmeugnt of as a testing activity. This is incorrect: 
Debugging starts with a known error and works for a 
Beorrection [ E] Becentiy, a Sioni eant body of 
literature and activity have been addressed to designing 
computer programs in a structured fashion in order to 
eliminate or minimize the occurrence of software errors 
[ 5, 10 ]. The theme of sone of these efforts is that if we 
design programs correctly through structured programming, 
there will be very little need for testing. Although these 
efforts do a lot to reduce the potential for errors, they do 


not act as a substitute for testing. 


Other testing activities include VON VA E ON 





validation, Gere if ication, proof of correctness, and 
performance testing. Hetzel [ 9 ] discusses these 
activities in relation to program testing. Verification is 
concerned with the progran's logical correctness based on 
execution of the program in a test environment. Validation 
is concerned with the logical correctness of a program in a 
given externai environzent. Certification implies an 
authoritative endorsement that a program is of a certain 
quality. A proof of correctness deals with the logical 
correctness without regard to the environment. Performance 
testing concentrates on the non-logical properties such as 
resource utilization. Each of these activities has much to 
afer. The problem arises when one of the approaches is 
assumed to equate to conplete testing. Lt rs clear that 
improved software quality must be approached from several 
fronts: improved design technigues, improved programming 
management, and improved testing methodology; rather than 


assuming that a panacea is present in a single approach. 


There are many funianental questions that must be 
answered in designing a test of an information processing 
system. One such question is what should be tested? Too 
often a tester ends up testing an incomplete or modified 
version of the system that is easier to test than the real 
System. Often the tester is faced with an infinite set of 
input combinations to be tested. In this case, the question 
becomes how can a subset of the test inputs best be selected 
to throughly test the systez? Another important issue is 
how should the test efforts be organized? It is important 
to get the most information about the system out of every 
Et run. ‘It is important to establish test data recording 
procedures at this time in order to insure that all error 


information will be recorded. This can be accomplished by 





properly organizing the tests in a logical sequence. Tests 
should be related CO Desi cana Sources of errors. 
Gruenberger [ 8 ] states that "part of the art of testing is 
knowing when to stop testing." This exposes a two sided 
question the test designer must face: whea is the test 
finished and what can be said about the system uhen testing 


is stopped? 


All these questions are futher compounded by the 
fact that there can be no set rule. Every system requires an 
Original test procedure designed to fit its special 
requirements. Gruenberger suggests "that the intellectual 
effort to test a program is of the same order as that which 
Created it."  Pernaps if ke nad said properly tested, using 
a minimum of resources, the effort would be on the order of 


the original effort squared. 


This thesis presents a test methodology that will 
help answer these questions. A model is presented that wiil 
Serve as a framework Cor the construction of a logical 


approach to system testing. 


BEEN A MODULAR APPROACH TO SYSTEM TESTING 


A modular approach to system testing offers many 
advantages for the design or the test and the development of 
the system. The modular aesign involves breaking a large 
System into many small Parts called modules. The 
intra-module functions are independent; however modules 
interact by means of standard interfaces. Each module 


performs a major function of the system. 


Modularity improves systen design. Modularity allows 


software to be portable. To an extent, modules may be 
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transferred among machines and operating systems. With 
Standardization of modules, they may be shared among many 
applications. With modules being shared in this manner, the 
programming effort is reduced and the reliability of the 
program is increased since the modules will be tested with 
each application. Tne progam may be expanded easier and 
changes are easier to incorporate since the effect of a 


change is localized. 


The testability of a nodular system is greatly improved. 
Testing of different nodules ray be carried out in parallel. 
Standardization of modules yields a set of assertions that 
may be used as test criteria for the modules. Hodules may be 
compiled separately anā can be stored in a program library 
and accessed independently. Modularity allows testing a 
system early in the construction phase. Each module may be 
tested as soon aS it has been constructed instead of waiting 
for the whole system to be completed before starting to 
test. Since modules car De reused, it will be economically 


feasible to do more testing. 
A modular system was chosen for the model in order to 


take advantage of these iuprovements in system design and 


system test. 


ol 
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Be THE FUNCTIONAL MODULE CONCEPT 


da Module Definition 
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In the representation of a system using the 
functional model, tne lowest element of the system that was 
considered was the module. Since the word module has had 
wide use throughout the conputer industry, it is necessary 
to completely define the desired application of the word in 
the model. A module is an entity that performs a function 
within the system. A function is an activity performed by 
the system such as a fast Fourier transform. The physical 
embodiment of a module is the Wiring and circuit board of 
hardware and the source or object programs recorded on 
punched cards or magnetic tape or resident in memory, for 
computer software. By addressing modules by the function 
they perform, a module is freed from the distinction of 
being just hardware or just software. The proper modules to 
test a system is determined by the accessibility of 
information within a system. In order that a module be 
useful for testing purposes, the module must have accessible 
ana controllable inputs and measurable outputs. Therefore, 
the modules would be chosen at the lowest level that the 


tester could ensure these properties. 


A module receives inputs and transmits outputs 
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across a boundary. A boundary consists of a location within 
the system at which the inputs to a module or the outputs 
from a module may be measured. In order for the tester to 
access these inputs or outputs the boundary must be 
identifiable. In order to accomodate this requirement for 
an identiable boundary, it is necessary to consider the 
composition of modules. The composition of two modules would 
be a module performing the same functions as the original 
two modules. An example oi composition of modules could 
involve two modules. One is a fast Fourier transform module 
and the other is a digital filter module. IE It. is 
impossible to identify a point to measure the output from 
the filter module to the Fourier transform module, the two 
could be considered as one module that performs the function 
of the filter and of the transform module. Thus, the system 
as a whole could be viewed as a module or a module could be 
conSidered to be a small unit of code. The proper level to 
identify modules will be indicated by the access to the flow 
of resources in the system anā the functions performed by 


the system. 


A module will be assuned to be free of internal 
errors for system test purposes. Internal errors are those 
types of errors which are round during unit testing of the 
module. This assumption is predicated on the fact that all 
modules vill receive extensive individual unit testing 
before the system is assenbled. If an error still exists 
within a module, the test system will detect it only as tne 
error applies to the module's relationship with elements of 
the system as a whole. Assuming that the test plan is 
surficent to detect all errors external to a module, the 
only way an error could go undetected would be if its 


- 


actions vere confined to the module itself. 


The system may now be described as a collection of 


modules which has external inputs and external outputs. The 
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selection of modules nust de such that every portion of the 
entire system is represented by a module and no portion is 


represented by more than one nodule. 


In performing this function, the mođule utilizes 
resources provided to the system. These resources may be in 
the form of data, control signals, or physical resources 
including both hardware anā software units. Thus a resource 
is an element of the system that is used by modules in 
performing a function of the system. Resources have tro 
types oi attributes. One type deals with the usage of the 
resource which is the anount or size of the resource that is 
assigned or available to be assigned. The other type of 
attribute of a resource deals with the contents of a 
particular resource such as the contents of a memory 
location or the actual value of a particular control signal. 
Resources have states. Tnese states indicate the status of 
the resource. Some examples of the state of a resource are 
E Ung, writing, idle, file empty, file half full, or 


memory region assigned. 


Dc Task Definition 


—— — wor a mg of mg e ee A ` me 


The work to be performed by a module may be 
represented as an ordered or random series of tasks. Tasks 
are the sup-functions performed by a module. A sub-function 
consists of the execution of a step in the algorithn which 
the module must execute in order to carry out its function. 
Thus, a module is the representation of a function within 
the system and tasks are the representation of the execution 
of the function. Examples of tasks are the computation of a 
Simple function, storing the result in memory and outputting 
Men result to the printer. This usage of the word task is 
syononymous with the use of the word "process" as it is used 


Burousmout the operating system literature. The precedence 
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of tasks is determined by the algorithm. It is also 
possible to have no precedence constraints. In this case any 
task could be executed whenever the resources were 


available. 


In order to execute a task, the module goes through 
a series of states. The state of a module is the status of 
the module at a given time. A partial list of states that a 
module can enter includes: compute, wait for memory, wait 
for input/output, wait for cpu, idle, input processing, wait 
for another module to complete a task, wait for a resource, 
and interrupted state. The particular state of a module is a 
function of the set of inputs to the module, resource 
States, and its previous state. The Enes of a module are 
a function only of the state of the module. A primary state 
is a state that a module is required to enter in order to 
perform a task. Primary tasks include compute, input 
processing and output processing. A secondary state is a 
state in which the module accomplishes no work. Examples of 
secondary states would be the blocked state, Wait for input 


nalt for cpu. 


The system state is the set of module states. The 


system state changes when any nodule changes state. 


3.  liodel Notation 


The following is a list of symbols used to describe 
the model, Each symboli is followed by the definition of that 


Synbol as it is used in this system of notation. 
12 —-— Hodule designation, 


CENE -- Current state of module i, 





* k ----- Next state of module i, 
A SZ Vector of inputs at module i when module 
Tt 
is in state j and input starts at tine ue 
Sa Pe Vector of outputs from module i after the 
ikt! 
i module has transitioned to state k and output starts 
at time t', 


kx TT = =e- mine at sA Ch ransitlon of module i from 


AT Ge CO State k OCCUrS, 


OA 757 Amount oz tine which module i spends in 
1) 
state j, 
IM <= 5et of resources used by module 1 when in 
1] 
State j, 
DEMO... Pl). 77 TT SS 
i n ij 


module i is in state j, 


AA St Time which module i uses n 
Voz hme 


resources when in state j. 


4. Model as a Directe 


Cep <p m au 


rey 
E 
r1 


aph 


It is possible to represent the system as a series 
of directed graphs. One graph would be required for each 
module. The nodes of the graph would represent module states 
the arcs would represent state transitions. Other 


information could be portrayed on the graph. The state 
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dependent information could be associated with the node. 
This would include the current state of the modulo, the set 
of resources used by the module in that state, the state 
vector for the resources used by the module, the vector of 
inputs to the module, the vector of outputs from the module 
and the amount of tine the module spends in the state. The 
arcs could be labelled with the time that the module 
requires to transition fron the source state to the 
destination state as is shown in Fig 1. Eno this figures 
the module i transitions from state j to state k at time 
T : 
ijk 
These directed graphs would give the tester a 
convenient means of visually representing the activity of 
the module. The tester migat prefer to show only the primary 
States of the module and the idle state instead of showing 


all possible states of the system. 


5. Time Domain of a 


A property of a module is that it uses the resources 
of the system only at certain times. One of the major 
problems of testing computer systems is trying to identify 
when two or more modules will be competing for the same 
resource or resources. The problem is further compounded by 
the system that uses multiple CPU'S which are running 
asynchronously. The concept of time domain vill be useful to 
address this problem area. A time domain of a module 
consists of the times that resources are in use. A graph of 
the time domains of the modules of the system would be a 
useful abstraction of the system for the analysis of the 
timing problem. The resources of the system could be 
represented on the vertical axis with time expanding along 


W-Whorizontal axis fron the orgin. Each area so represented 
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Pig 1 


DIRECTED GRAPH OF 
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MODULE STATE TRANSITION 





should be labelled with the module and the amount of the 
resource required. The tine domain of a module would be the 
summation of the area representing the resources used by the 
module at any given time. Any intersection of time domains 
would represent a potential error only if the total demands 
of the modules exceed the naxisum resource available. It is 
also possible for one module to exceed the maximum resources 


available. 


One problem with this representation of the system 
is finding a timing system that applies to all modules when 
the system has modules working asynchronously. In this case 
the time axis would be the elapsed time from some critical 
event that appears throughout the system. The changes in 


System state will be referenced to this event. 


It iS possible to consolidate the represention of 
module states into a system state representation and show 
resource usage confiicts in terms of system states as 
indicated in Fig 2. In this figure there are five types of 
resources available to the system. They are labelled R1 
through R5. The amount of each resource is indicated on the 
vertical axis. For example, there are six units of R2 
available. There are two resource conflicts  protrayed in 


this system. One occurs in system state ET Here one 
module requires four units of resource Rü and another module 


requires two units of R4. Tne conflict occurs because there 
are only four units of R4 available. The: Cont laeer us 
denoted by a cross-hatched area. The other conflict is in 


Systen state y A mođule has requested six units of 


resource R3 when only two units are available to the system 


| NEP 


The construction of such a graph would be infeasible 
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Bodo by hand for a real system. A program could be written 
to produce this type of graph £rom the time domains of the 
modules. On this graph the computer could identify resource 
usage conilicts. The program would be independent of the 
Systen being tested; therefore, it could be used on many 


different projects. 
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BEEZEHPPLICATION OF MODEL IO TESTING 


1. Functional Specifications 
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One Ene nore difficult processes in producing 
viable software is translating the design characteristics of 
the system into meaningful requirements for the programmers. 
Penn, McClean, and Urfrig { 2 |] vividiy demonstrate the 
Magnitude of the problem in their study of a large software 
project. The authors divided errors into two classes. These 
were design errors and coding errors. An error was 
considered a design error oniy if its correction caused a 
corresponding change to the design specifications. Of the 
total errors, 64 percent were design errors. This alone is 
enough to illustrate the need for a valid method of design 
Specification. Even more disturbing was the time frame 
within the testing that tbe errors vere discovered. Of the 
54 percent tnat were not discovered until the acceptance, 
integration or delivery phases of testing, 45 percent were 
design errors. The remaining nine percent were coding 
DOES. Errors discovered in these latter stages of system 
design would be much more difficult to correct than ones 
discovered early in the coding of the project. If the 
Majority of errors are going to be of the design type and if 
these are going to be the more difficult type of errors to 
correct, it is necessary to have a good system of describing 
the design specifications LO the programmers. The 


functional model is an attempt to build such a syster. 
When the functional model is used, the user should 


be required to define all functions of the system. The 


functional  reguirements would consist of a statement of the 
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activities of the system and the associated inputs and 
outputs. By requiring these functional requirements, the 
programmers are assured of having a complete detailed 
description of the system at the beginning of tne project. 
This description should aid tie programmers in thelr 
understanding of the overall requirements of the project and 
provide sufficent detail to properly code the project. This 


would reduce the number of design errors that would occur. 


It is possible to over-specify the design of a 
Systen. This couid prevent the programmer from choosing the 
most efficent method of coding tne project. It could also 
introduce errors into the system, if the user does not have 
a in depth knowledge of computers. This trap is avoided 
using the functional requirenents of the model. The details 
are presented in terns of functions of the system which is 
the area in which tne user is most knowledgeable. The 
implementation of the functions is left to the programmer, 


who is in a better positioa to determine the proper method. 


Another pitfall of system design may be avoided 
using the functional requirements. Frequently, test 
specifications are not available early in a project because 
testability is not considered to be a design parameter. 
Instead, test reguirements are fornulated as an afterthought 
when it is too late to influence the design [ 17 ]. 
Functional test specifications are defined as test 
specifications which are based on testing the stated 
functional requirements and observing the corresponding 
outputs of the system. Functional requirements should be 
incorporated in the functional test specifications. 
Detailed design should not commence until this information 


is available. 


2. Documentation 





The need for coaplete and usable documentation 
should be a primary concern with anyone involved with any 
phase of system design aná testing. Poole [ 13 J implies 
Maat the lack of good documentation usually means that 
testing is not performed as thoroughly as it should be and 
debugging is that muca more complicated." Another 
application cf documentation is the maintainance of the 
system. Since the life cycle of a system iS much longer than 
the development phase, the designers will probably not be 
available to help maintain the system. In addition, many 
different people may have aaa access to the software of the 
system. Any changes made sy these people must be preserved 


for future use. 


The functional zoiel attempts to provide adequate 
documentation throughout tae life cycle of the system. The 
concept is to force docuzentation to be an integral part of 
System development. Two documents have already been 
discussed. These are the functional requirements of the 
system and the functioaı test Specifications. These 
documents should form a sejaent of the documentation. These 
Should be systematically updated as changes are made to the 


System. 


The documentation snould include other information 
as well. This could include a data base containing 
information about all errors that were found in the system 
during the life of the systen to date. Unfortunately there 
is a tendency to ignore this aspect and to think of this 
type of information as soaething to discard once the error 


nwbeen corrected [ 13 ]. 
Every incident aust be recorded because an error 


that may appear insignificant to the user could be an 


inportant indicator once it is properly analyzed. The data 
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base could be used to classify modules that are the source 
the majority of errors. This classification could be used 
eai rect future testing and debugging. It could also be 
used to determine wnich modules are the most unreliable. 
IS would provide a starting point if one desired to 
improve the reliability of the system. This would be 
particularly applicable if the module that iS moSt critical 
to the system's operation is also the most unreliable. The 
data base could also be classified as to type of errors. 
This would be valuable information when designing a similar 


System. 


Another FO iM oí documentation that should be 
incorporated into the plan for system testing is assertions. 
These are statementS that are introduced into the code by 
the prograrmer. These state a fact about the design of the 
program. These statements nay be treated as a comment card 
or used to prođuce code to check for the validity of the 
assertions. The appropriate action would be determined by a 
parameter passed to the conplier. Two types of assertions 
could be employed within tne model. The first would be 
global assertions. These would be En the form of 
specifications for  interaodular actions of the system. An 


example of such an assertion would be 
ASSERT RANGE OF ALL ARRAY INDICES IS 0 TO 100. 
The other level of assertions would be local. The local 
assertions would be defined by the programmer but within the 
design specifications. An example of a local assertion 


would be 


DESERTENANGE OF I IS 10 TO 20. 


These assertions would be a permanent feature in the 
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Meogran. They could be activated on the local level to help 
test a module or on the global level to aid in introducing a 
change to the system. As such, these assertions would fora 


an important part of the systea documentation. 


- 


3. Test Inputs 


Ideally, it would be proper to exhaustively test a 
System. This implies that every path in the logic of the 
program be executed and tested. Shooman [ 12 ] demonstrates 
that this will normally be impossible due to the large 
number of inputs required. The problem presented involved 


exhaustiveiy testing an assenbly language program which 


2 
solved for the roots of a quadratic equation Ax +Bx + C = 


0. The computer was assuaei to have a 12 bit word length 


and integer arithmetic was used. All syntactical errors had 
been elimated and all known special cases suchas A - 0 and 
imaginary roots had been cared for, The input space to 


9 
exhaustively execute tnis program involved 64 X 10 


combinations of A, B, and C. The program had a run tine of 


240 microseconds per execution. The tine to complete the 
entire execution of the program over the input space was 
Bol hours. To test the program, the solutions must be 
verified by some independent means such as a desk calculator 
or a different algorithm. This should be done in as many 
different ways as possible, since there is a probability 
that two independent solutions will compute the same wrong 
Solution. Obviously, exhaustive testing is infeasible for 


even a small program. 


The problem tne tester must solve is how best to 
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select tne subset of test inputs from the universe of 
possible inputs. A metnod for selecting the inputs for a 
test is to first identify and rank the modules in a system 
by the criticality of the nodules to the mission success. It 
1s seldom the case that all nodules are equally valuable. A 
technigue fcr determining criticality is to ascertain the 
consequences to the mission of a module malfunction. A 
Malfunction in some modules would cause a mission abort, 
While others would result ina degraded mode of operation. 
The modules are ranked according to criticality. The time 
spent in testing each module can then be allocated uSing 
this ranking. The time allocation can be further refined by 
meerng the criticality of each sub-function of the module. 
This would be based on tae criticality of the sub-function 


to the performance of the function by the module. 


There are other factors that can be used to rank 
modules for testing purposes. One such criteria would be 
forcasted errors. Schneidewind [ 16 ] has developed a model 
of the occurrence of errors detected during functional 
testing of command and control software, Based on the 
results of a model that has characteristics similar to the 
one being tested, it would be possible to rank the modules 
in order of forecasted errors. Work is progressing in the 
area of developing relationships between program structure, 
program complexity and the ability to detect errors in a 
program [ 3 ]. Another netnod of obtaining such a ranking 
would be through the use of simulation. Critical modules 
could be identified by their high rate of failure in the 


simulation. 


Once the amount of testing resources allocated to 
each module has been determined, the proper number of inputs 
for testing each module can be fixed. The problem then 
becomes one of selecting the best inputs to throughly test 


each module. The module represents a function which maps 





the set of inputs into the set of outputs. The inverse 
correspondence could be used to obtain the set of inputs. 
Given this set of inputs, select test cases in order to 
cover the set as thoroughly as possible. Pay particular 
attention to the inputs that are involved in the control 
flow of the program. Once this has been done and if there 
are test resources still available, investigate unusual 
cases that may cause the system trouble. A possible source 
of unusual cases would be indicated by the set of inputs. 
Pick values that are combinations of the extremes of the 
ranges of inputs. Investigate combinations involving zeros. 
This 1S an excellent chance to investigate some of the "what 


muwEcuestions. 


4. System Representation 


Having fully developed the notion of a module, it is 
necessary to investigate the method that will be used to 
represent a system as a collection of modules. A system is 
comprised of asynchronously operating application software 


modules, hardware modules and executives. 


A system may be represented by the model. Pig 3 
gives a generalized representation of a processing system. 
The system represented in this figure is conprised of two 
asynchronously operating executives, A and B. These are 
connected to two seperate control buses noted by Control 
Traffic Bus A and Control Traffic Bus B. Each bus connects 
the application software modules and hardware modules that 
are controlled by the executive on the bus. An example of 
the traffic on this bus is a hardware generated interrupt 
Exsng at the conclusion of an input/output operation. A 
subsystem 1S comprised of one executive, the modules that it 
controls, and the control bus connecting the modules to the 


executive. There is a message traffic bus connecting all 
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modules. An example of the traffic on this bus is a module 
passing a computed value to another application module. 


Eternal inputs and outputs are identified. 


This representation of af system has many useful 
applications to testing. The model may be used to verify the 
merrect UNE tE Lona y a two types OE intermodule 
communications. The first is message traffic. The traffic 
on the message bus could be checked against tne functional 
test specifications for correctness. The second concerns 
@emerol traffic. The traffic on the control buses could be 
checked in a Similar manner. Other problem areas that could 


be investigated using the nodei include: 


* Are the various state transitions possible, based on 


the values of tne resource states? 
* Are there any blocked or deadlocked states? 
* Are the amounts of time in each state excessive? 


* When a module state transition occurs, are the 


resource stete vectors correct? 


nre the tines that a moduie holds resources 


excessive? 
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INI. SIMULATION 


RA SIMULATION OF THE XODEL 


A simulation of the model vas constructed. The 
simulation was an event store type of simulation. It was 
writtrn in FORTRAN to run on the Postgraduate School's IBM 
MOT. The simulation used the model representation with 
the user providing a description of the system to be 
simulated. This description required the nunber of modules, 
the number or tasks, the precedence among tasks, number of 
resources and resource us2ge. A complete description of the 


simulation appears in Aprenaix A. 


The simulation showed that the model could represent a 
System. A Simulation of this nature could be useful in tne 


Pest environment. 


RED LG AT LION IN THE TZST NVIRONMENT: 


IN Iuvestigation 9f rizing Problems 


—- — — gg n wé zm — —— n un ao e mmm. gef oe nie ee A A — ` 


Timing problems are extremely hard to investigate in 
a real system due to the fact that any test equipment 


installed internal to the system disturos the timing of the 
i 


EE Such equipment installed external to the system thay 
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not be able to gain the required information either because 
of synchronization probleas or because of access problems. 
By using a simulation o£ the system, the tester may observe 
any timing parameter desired. The tester may even ada 
timing parameters that are not found in the real system, but 
are necessary to completely evaluate some time critical 
Operaticn. By doing this, the tester may observe timing 
problems that could not be observed on the real system. 
This addition could be made without disturbing the timing of 


the real system. 


Ancther problem area of testing that cculd be 
investigated through the use of a Simulated system is the 
reaction of the systea to various rates of inputs. In the 
simulation of the model, it was possible to vary the mean 
time between arrival of inputs to the system. This parameter 
could be decreased on each run to determine the maximum 
input cate that the system could receive and still process 
an acceptable amount of the inputs. Another method would be 
to plot average tine to process a complete input verus input 
rate. This graph could be used to determine an acceptable 
range of input rates for the system in order to completely 
process an input in a given time period. This method of 
anaylsis could be used «hen testing a systen that has to 
produce outputs on a regular time basis, such as a system 
using a graphics display that nas to be refreshed at a given 
rate. Another important use of this method of analysis 
would be to determine now much the input rate could be 


mmereased in the future. 


If the time that a module spends in a particular 
State is expressed as a range instead of a constant, a 
Simulation would be an invaluable aid to the tester in 
investigating the operation of the system. One approach 
would be to observe the overation of the simulated system 


with all modules functioniaj at the maximum time range. 
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Another method would be using various combinations of module 
Operation times, selected frow the ranges, to determine 
under what circumstances the system would fail or be in a 
degraded mode of operation. This method would give the 
tester tne ability to checx the system over ali combinations 
of tine ranges to prove the validity of the range and to 
mMmeentify critical areas within the ranges. This can easily 
be done ón a simulated system out could be impossible to do 
on a real system because the tester wouid be unable to 


control the time the module spent in a state. 


Another timing problem facing the tester is the 
timing of the system as a whole. Often the tester would like 
to slow the system down or perhaps speed it up in order to 
Observe some particular action of the system. This would be 
important if the tester was unable to measure the output of 
a particular module because another output arrived before 
the first output could be measured. In a real systen, it 
may be impossible to change the timing of each component of 
the system by the sane amount. This would be particularly 
difficult in a multi-executive system. With a simulation of 
the system, the tester would be able to adjust the timing of 


the system in any manner ne saw fit. 


Some problems do not occur until the system has 
processed a large number of inputs. The tester may not De 
able to cycle the real system through such a large number of 
inputs due to time or some other constraint. However in a 
simulation, the time scale may usually be greatly compressed 
allowing the tester to cycie the system through many times. 
This would greatly increase the probabilty of discovering 


weent bugs. 


2. Fault Insertion 
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DwWetral 6 ] contends that "testing «can only 
determine the presence of errors, not their absence." One 
approach to try to prove the absence of faults would be to 
know the reaction of the system to every possible error and 
combinaticn of errors. Using this knowledge, one could 
simply observe the reaction of the system and state what 
ok 5 were Or Vere not present. Unfortunately, the set. of 
every possible error, combination of errors and system 
reactions is an infinite set. Therefore the above approach 
would also fail to prove the absence of faults. However, 
the approach could be used to greatly expand the subset of 


errors that the tester could detect. 


The tester may introduce a fault in the system. The 
reaction of the system to this fault could be catalogued for 
later reference. This information could be used to identify 
modules that are affected the most by a class of errors. 
This set of modules would be noted for special testing. This 
information could also be used to ensure the validity of the 
Mest plan. If the group or tests indicated in the test plan 
did not encompass the reactions observed in the simulation 
then the test plan would not be able to detect that 


Euwticular fault, 


Far tial System Siaulation 
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Frequently the tester will not have the time, 
assets, or motivation to perform a simulation of the entire 
system to be tested. In this situation, simulation of 
certain parts of the system may be desirable or the tester 
could choose to simulate the system in less detail. Campbell 
Ome Heffner [ 4 } relate a case history illustrating this 
point. A Simulation model was constructed of a system being 
developed. The skeleton system was working before the model 


was debugged. When the model was finally working, no one was 





sure which version of the system the results were meant to 
represent. However, some of the designers used simple 
Simulations that they developed to study certain aspects of 
the system. The authors concluded that "ambitious 
large-scale models generated by professional model makers 
are less helpful than simpler work done by the system 
developers themselves." Obviously a simulation with less 
detail was more useful in this case than a complete 


simulation. 


Quite often in prototype testing, a module or module 
will not be present when the tests are scheduled to 
commence. This could be due to late delivery or to a module 
being modified after prelininary testing proved the module 
needed modification. This could also be caused by a planned 
action such as phased delivery. A simulation of this module 
would ailow the tests for the rest of the system to 
continue. Simulating the missing nodule would be 
particularly easy if the system had been described in the 
form of a functional model. if all the information required 
to functionally represent the model is present then a 


bon can be constructed from this information. 


Another use of simulation involving less than the 
whole system is the test data generator. When the system is 
being tested in the laboratory environment, it may be 
necessary to simulate the inputs to a system. Since there is 
no reason to believe that all modules will be present during 
the entire test phase, the tester may desire to have the 
test data generator be able to simulate the output from any 
module. Thus the test data generator could also replace any 
missing module as the system was being tested. Not only 
would the generator have to act as a output generator but it 
must also act as a destination for outputs from other 
modules intended for the missing module. If these outputs 


were not accounted for, the reaction of the partial system 





would not be truely representative of the reactions of the 


real system. 


Mmeecer having spent much time and effort to develop a 
Simulation, the user may find tnat the simulation addressed 
the wrong problem or worse yet solved no problem at all. 
The validity of a Simulation is the consistency between the 
Simulation and tne real system it represents. Proof of the 
validity of a simulation is alaost impossible, especially if 
the real system has not yet been constructed. By the time 
the real system has been constructed and the validity of tne 
simulation is disproved, irrevocable decisions may have been 


made based on test data froa tne Simulation. 


Although validity is a major problem in simulation, 
it is by no meanS the only problem. A list of problem areas 
that may cause a misunierstanding of the system being 
Simulated 1s presented in Fishman [ 7 ]. These include 
input parameter misspecification, influence of initial 
conditions on data and misuse of estimates. Along with the 
list, there are suggestions on ways to control these 


problems. 


Prototype testing may result in design changes. 
Beenechange requires a change in the model. If the tester 
has not allowed for such an ocurrence in budgeting his 
Station resources, the nodel would represent a system 
totally unlike the real system. Also, there would be a tine 
lag in modifying the model. This could have a serious 
EN ct On the test schedule if this contingency is not 


included in the plan. 
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À. APPROACH 


ill. Test Plan 


ae — => € o 


oO caer"” toscapply the functional model to the 
problem of system test it is necessary to develop a test 
plan. A test plan should be created as part of the design 
plan. As a minimum the test plan should discuss the 


following major elements of tne test: 
* define modules 
* define nodule states 
* identify inputs and outputs for each module state 
* identify module interfaces 
* define resources and resource states 
* identify resource usage for each module state. 


The test plan must also include the functional requirements 
and functional test specifications. In addition it should 
include the test procedures. This would identify acceptance 
criteria, such as the allowable divergence between desired 
aná actual output values, tine duration of tests, allowable 
number and types of malfunctions, number and distribution of 


test replications, and means of checking test results. The 
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test plan should identify major testing milestones. These 
would identify major sections of testing that must be 


completed before systea development can continue. 


The test plan snould document the subsystems ira 
will be tested. This vili require the development of a 
method of isolating a subset of the system to test it 
without the effects of the remaining system being 
mt roduced. These identified subsets of modules are called 


subsystems and will be used to test the system in stages. 


Besides the modules ina subsystem, the test plan 
must also define a set of zeasurements which will indicate 
wheather correct cutputs are being produced for given inputs 
and define the hardware and software locations of the 
measurements. The plan nust describe how to instrument the 


system in order to obtain these measurements. 


The test pian should develop some organizational. 


EUucture. This would inciude who is to do the testing and 


the resources to be used in testing. The plan should 
include who is responsible For maintaining the 
documentation. This wouid include the test data, the error 


information, design changes and test modifications. 


Æ Subsystem Testing 


m rn o cd i in i CANE pesi 


The modularity of the model allows the user to commence 
testing as soon as possible. This will reguire the testing 
of a subsystem . The subsystem is defined in the test plan 
and testing will conmence aS soon as all the modules of the 
subsystem are available. This is illustrated in Fig 4 In 
EUN figure there are two application modules in the 
subsystem that are ready tc be tested. The oniy hardware 
ED is required for the functioning of the modules is 


Hardware Nodule One and Eardware Module Two. These are 
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connected to the application software modules by a test data 
generator. This is a prograa that will simulate external 
inputs and message inputs iron missing modules, The set of 
external inputs and input seyuence is based on typical 
operational scenarios and tne critality of the various 
modules to mission success. another required program is the 
control input Signal generator. This works in conjunction 
With the executive and generates simulated control inputs 
for the missing nodules. In some cases this program coald 
be developed to replace the executive itself during early 
testing, when the executive may not be available. The 
locations where input/output measurements are made are 


identified. 


The system under test will be expanded as testing 
proceeds. When the next application subsystem becomes 
available, it and the hardware it uses could be added to the 
system. The subsystem could consist of a single module or a 
group of modules. The intent is to test the system in 
Stages starting with the minimum number of modules, and 
increasing the number of aodules as testing progresses, 


until the whole system is present. 
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This is the only hardware required for 
functioning of application modules one 
and two. 


Fig 4 SYSRKENR IS EACONETGURATION 


49 





S CONCEUSTONS > 
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There is a great need for better organized test methods. 
Ee the impact of computers on society increases, stringent 
requirements will be iaposed on the reliability of computer 
Systems. Wien Missiles that can destroy Ma coumtry sarc 
Ep usted to the control of a computer, even one error is 
unacceptable. Recent cevelopnents for designing and testing 
computer hardware have ione such to improve the reliability 
of systems. Unfortunateiv, development of software testing 


has not kept pace. 


A model was presentsi that may provide a means for 
Metacking the problen cz testing a large software 
meveloOpment project. The strength of the model is based on 
two concepts: modularity 2nd functional representation. The 
benefits offered by a moiular approach to testing have been 
discussed. Modularity appears to offer a viable solution to 
many of the testing problezs. The functional representation 
concept is crucial to the development of a sound testing 
policy. The user must De given an opportunity to specify 
exactly what the system is expected to do. The tester must 
test against these specifications and not some artifical 


System. 


A notation was provided for the model. The notation 
allows the tester to describe the attributes of the modules 
in a compact manner. The aodel was analyzed as a directed 
graph and the notion of a time domain of a module was 
introduced. Both of these concepts should prove useful in 


Eus a system. The application of the model to testing 
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was explained. It should be emphasized that this system is 
not intended as a unalterable system that provides the 
solution to all testing problems. Rather, it is a beginning 
point for the development of a system test methodology that 


Will be improved each time it is applied. 


A simulation of the nodel was constructed and was 
described in the appendix. Uses of simulation in the test 
environment were discussed. Simulation has much to offer 
during the testing phase of system development. One of the 
difficulties encountered was construction of the simulation. 
There does not appear to be a good special purpose 
Simulation language or system for testing. An interesting 
future development would be to expand the model. The system 
would permit the tester to describe the system to he 
deveioped. The system should contain a package of testing 
tools, such as debug and restart capabilities, that the user 


could invoke as needed. 
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APPENDIX A 


DESCHRPETION DESSTAULATION OF NODER 


1. Definition of Teras 


The following terns were defined in the text: state, 
module, task, and primary state. Their usage in this 
description oi the simulation of the model will be 
consistant with these definitions. The status of a module 
was defined to be the information required to identify a 
particular module, and its state, that is executing a task 
for a given input. Thus, the status of a module identifies: 


module, state, task, and input number, 


All times in the program were based on the standard 
time unit. The standard tine unit may be defined as the user 
desires. The only requirement being that the user must be 
consistant throughout the program and that all times be 


integer nunbers. 


mine simulation of the nodel was an event Store type 
Of simulation. In this type of simulation the time is 
advanced each time an event occurs instead of on a fixed 
interval. This prevented the occurrence of a large number of 


intervals during wnich there was no activity. This type of 
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Simulation maintains the complete status of the system 
during the simulation because events are executed in the 
order in which they occur. The events in this simulation 
consisted of the arrival of inputs, the completion of a 
primary state, data collection events, and the termination 
event. phecsevent that  represeuted tne cofipletion of a 
primary state was always followed by the occurrence of the 
next primary state unless the input terminated or the 
resources required to enter the next primary state were not 
available. A data collection event was a time period during 
which the user could measure any parameter of the 
simulation. These occurred at regular intervals taroughout 
the running of the simulation and could be used to gather 


data to compute such statistics as average queue size. 


The programming language chosen for the simulation 
language was FORTRAN. The motivation behind this choice was 
to ensure the portability of the program. FORTRAN is not an 
ideal programming language for simulation. Other languages 
such as GPSS (General Purpose Simulation System} provide 
many of the building blocks needed for the simulation such 
as the clock and the calendar. Although FORTRAN does not 
provide these blocks, it is readily available at most 
Ms titutions. Another limit to the portability of the 
program is the random nunber generator which was used in the 
sinulation. The program used the LLRANDOM package available 
at the W. R. Church Computer Center. The random numbers 
were passed by subroutine calls. By adjusting these calls, a 
new random number generator may be introduced into the 


progran. 


The simulation vas designed to be able to simuiate a 
wide range of computer systems. This was implemented by 
MiWloving the user to input various characteristics of the 
System. These characteristics are discussed in the input 


section of the appendix. The resources used were identified 
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throughout the program oniy by number. The user was to 
assign both the type of resources the system uses and the 
amount of each resource available to the system. the only 
requirement was that all quantities were to be integer 
values. The workload for the system was defined as a system 
Ee tasks. The user was required to specify the priority of 
tasks. Modules were identified by number. The user was to 
specify which module or modules was to execute each task. 
The user waS also to Specify which resources and primary 


states were reguired in the performance of each task. 


The simulation was programmed in a modular fashion. 
eeni major function of the simulation consisted of a 
separate subroutine. This design was chosen to facilitate 
any modification of the program and to improve tae 
comprehension of the program by another user. All major data 
structures and variables were made available to all 


subroutines by the use of coumon blocks. 


=- á — m gun 


The simulation allowed the user to vary the input 
parameters listed below. The description is in the order in 
which the inputs were required by the program. If no range 
is given the parameter did not have limits placed upon it by 
the program. These linits were caused by the size of the 
data structure used in the program. The user could have 


chosen any reasonable value for the unlimited parameters. 


Subscripted variables indicate multidimensional data 
structures. 
x MAXIN --- Determined the number of inputs the system 


May process at one time. Range was one to ten inputs. 


* MEANIN --- Represented the mean interarrival tine 


for the system. Units had to be standard tine units. 


45 





NUMRES --- Total nunber of resources required by the 
System for proper functioning of the system plus one 
additional resource. The first resource was used by 
the simulation as a time representation. Therefore, 
one additional resource was required. This allowed 
the user to specify the amount of time the resources 


were in use. The range of NUMRES was from two to ten. 


MAXRES(1) --- This vector gave the maximum amount of 
each resource available to the system. The units were 
unconstrained except time units had to be in standard 


time units. The range of I was from one to NUMRES. 


NUMTSK --- This was the number of tasks assigned to 


the system. The range was from one to ten tasks. 


PRECTSK (I,J) --- Tkis matrix was used to give the 
precedence relationships among tasks. Each colunn was 
identified by a tasx. This task blocks all tasks 
listed in the coluan. The first element of each 
column was the number of tasks blocked by the given 
task. For example, ii column number three contained 
the following nunbers reading from top down: 2, 4, 6. 
This would indicate task number three blocked two 
tasks, namely task four and task six. The variable I 
had a range from one to NUMTSK. J was confined to the 


same range. 


MAXSEO --- This was the maximum number of primary 
erstes that a module had to go through in order to 


execute a task. The range was one to eight states. 


DOS (E, J) --- This matrix showed the different 
sequences of primary states that a module was 
meqg@ised to go through to complete one task. Dach 


row represented the path for one module to perform 
one task. The first element of a row Was the task, 


the second elenent was the module, and the remaining 





elements were the sequence of primary states. A zero 
indicated the sequence had been completed. The 


variables J and J had a range of one to ten. 


* ENDIR --- This was the number of different primary 
States for every module, The maximum value was 
ERE Y. 

* TSRKRES (1,3) --- This matrix gave the amount of each 


resource required for the execution of each primary 
state of a module aná the associated task. Each row 
Was the resources recuired by one task/state/module. 
Thus the first three elements of each row represented. 
the respective task, state, and module associated 
with the resource requirements given in that row. The 
first entry represented the time that the resources 
would be required in that state. The variable I had 
a range from one to zNDTR. The range of J was from 


conecto NUNKES plus three. 


The following parameters were assigned values in the 
Start subroutine of the program and may be varied as the 


user desires: 


* MAXTIM --- This variable gave the total time the 
simulation was to be run. Units were standard time 


units. The value assigned was 1000000. 


* DATINV --- This variable was the time between data 
collection events. Units were standard time units. 
los value assigned as .2500.The user was Lreelco 
choose any value for this parameter. However, the 
value chosen should be realistic in terms of the 


other times used in the program. 


* IX --- This variable was the seed for the random 
number generator. It could have had a value between 


1 ander 7 483647. A value of 47979951 was used. 





This variable was designed as an input to the 


LLRANDOM randoa number generator. 


System parameters were built into the simulation and 
were located throughout tae progran. These system 
parameters may be changed as the user sees fit but with 


Mater difficulity than the ones listed above. 


* Number of tables --- One system table was used in 
which the status of all modules awaiting resources 
was held until the resources needed for a particular 


State were available. 


* Size of table --- The maximum number of module Status 
entries that could De stored within the system queue. 


This was assigned a vaiue of thirty. 


* Service discipline --- The table entries were 
serviced in a first-in-first-out manner, based on 


time entered in the table. 


* Resource allocation policy --- A module task was not 
executed until all resources for that module were 


available. 


* Probability of a task --- Task branching was done on 


an equal probability basis. 


4. Data structures 


The data structures which were used in the 
Simulation were divided into four main classifications. 
These were the system table, the simulation calendar, the 


event records and the utility group. 


The system table was represented by a group of 


vectors. A vector was a one dimensional array that was used 
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Store. Status information. The group of vectors were 
Meeessed by a common index. The index was stored on a 
nate vector in order of request for resources. Thus a 
komplete description of any module waiting for resources 
Seuld be obtained without a sequential search of the table. 


This facilitated the user Specification of service 
discipline. 


The simulation calendar was represented by a group 
of vectors in the same manner as the system queue. Besides 
the information describing tne status of modules that were 
in service, this group also contained vectors representing 
simulation information such as event type and event tine. 
The event time was tne time representing the scheduled 
completion of the event. The event type differentiated among 
end Of state events, data collection events, arrival of 
input events, and termination events. A zero in this vector 
represented an empty slot on the Simulation calendar. Each 
time an event was selected the event time vector was 
searched sequentiaily to determine the next event occurrence 


We in the simulation. 


The event records represent a diverse group of data 
WWilctures used to preserve information about the events in 
the simulation. The generai form of these data structures 
were matrices with each row representing one input that was 
being processed by the system and a column for each data 
element to be preserved for each state. The data elements 
were items such as time input entered system, time module 
used resources in a primary state, and the order in which 
primary states were executed. In order to avoid being 
overwritten, these matrices had to be printed out at the end 
of each sequence of tasks. The column index "to Zzenese 
Matrices were all based on the number of state transitions 
and each index was adjusted for the proper number of 


elements being preserved in the matrix. 
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Utility records were used to save information as it 
was being moved fron the system gueue to the simulation 


Calendar. 


= SS oo —— 


Two types of output were provided for in the 
simulation of the model. These vere event reports and 
sequential reports. The event reports occurred shenever the 
event with which they were associated was terminated. Three 
types of event reports were used in the 'simulation. These 
were end of state report, end of task report, and 


termination report. 


The end of state report occurred at the end of every 
primary state. The report contained the status of the 
module, The report gave timing information about the 
primary state just completed. This included the time the 
module was placed in the taple, the time the module entered 
service, time the module finished service, total time in the 
system, total time in service, and total time in table. The 
report showed the amount oí each resource available after 
the resources used in that state vere returned to the 
System. A printout of the system table was shown after the 
next module status had been added to the table. All modules 
that were scheduled at that tine vere reported giving the 
status of each module. The report concluded with the status 
of all system resources after the scheduling activity had 


been completed. 


mee end of task sequence report occurred after the 
complete sequence of tasks for one input had been completed. 
The first part of the report gave the sequence of states, 
identifying each state by task and module. The second part 


of the report gave information about the timing of the task 
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Sequence. This included the time the first module entered 
service, the time the last nodule finished processing, the 
total time required if the sequence of tasks had been 
processed serially, and the total time the input was in the 


System. 


The termination report gave the total number of 
inputs that were lost by the system when it was already 
processing the maximum nuzber of jobs allowed in the system. 
This report could be used to give averages from the data 
collection events such as average queue size and average 


percent of each resource being used by the system. 


sequential reports were not used by the simulation 
but could have been initiated by the data collection event 
and used to give an instanteous picture of the system at 


regular intervals. 
6. Program Flot 


The program commenced with a call to subroutine 
Start. This subroutine initialized all the variables and 
returned control to the main program, The next event was 
selected by the subroutine NEXT. This subroutine selected an 
event based on a sequential search of the simulation 
calendar. Control was returned to the main program which 
called the appropiate subroutine to handle the selected 


ewent. 


If the next event waS an arrival of an input, 
subroutine NEWINP was called. Subroutine NEWINP inserted 
the first module status on the system table with a call to 
Mme subroutine PUTONO. this occurred only if there was space 
in the system for another input and iz there vas room in the 


system table. If there was no space in the system, the input 
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NES counted as a lost input. If there was no room on the 
System table, the progran was terminated with an error 
condition. With the exception of an error condition, the 
next arrival of an input was placed on the simulation 
enmendar with a call to the subroutine ENTER. A call was 
then placed to the subroutine SKED. This subroutine checks 
to see if the current available resources of the system are 
such that a module on the system table may commence service. 
All modules were examined on a first~come-first-served basis 
and aS many as possible were placed in service. The end of 
State event was then entered on the the simulation calendar. 
Controi was then passed to the main program where the next 


event was again selected. 


The subroutine ENDSTT was called. The resources used 
by the module were returned to the system. If the completion 
Of the state did not conplete the task, the next module 
Status was placed on the table and subroutine SKED was 
called. If the state conpleted the task the next task was 
selected. The first module status of this task was then 
placed on the system table and again SKED was Called. In 
either case STTDUMP was called to produce the end of state 
report. If the task was a terminating task, subroutine 
DUMPIN was called to produce an end of task sequence report. 


Control was then returned to the main progran. 


Two other types of events could occur. These were 
the data collection event and the termination event. A data 
EN ection event resulted in a call to subroutine WHAT. This 
subroutine collected the assigned information, scheduled the 
next data collection event, and returned control to the main 
program. The termination event occurred at the maximum 


simulation time and produced the termination report. 


7. Desirable Features 


vati ër gen a ie cò id ed E nn «= «rs sò ee A A mom 
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The generality of the simulation and hence its 
applicability to any real system was restricted by building 
choices for parameters into the simulation instead of giving 
the user a choice. These parameters included the queuing 
bë Gipline, resource allocation policy and the distribution 
of time spent in a state. In an ideal simulation of this 
nature, there would be many choices of these parameters, 
with the user having the ability to specify the discipline, 
Bey, or distribution that best represented the system he 
intended to test. 


During t he development of the program, other 
features that would have added to the usefulness of this 
type of simulation were Giscovered. One such feature was a 
debug package. This could include such items as a stall 
alarm. This alarm would give a warning if a module was in 
the system table for an excessive time period. The length of 
this time period could be selected arbitrarily and then 
adjusted by the user based on the system description. This 
alarm could be used to detect modules that were deadlocked 
or to set a maximum time that a module could be in the table 
before a failure occurred. Beizer [ 1 ] indicated various 
other alarms that would appear useful in testing the 
sinulation. These include watchdog alarms, cycle alarms, 
improper sequence alarms, and nultiple or missed interrupt 


alarms (if the user chooses to install interrupts). 


Another feature that would be useful in the testing 
environment would be a restart capability. The status of 
the system at the termination of the Simulation could be 
saved in a dump file. The user would then be allowed to 
stop the system and make any observations he desired and 
meme start from the same point. This ability would be 
particularly useful in investigating errors that occurred in 


random fashion. 
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TnoMebowe feature could be best "utilized as 3 
interactive program. The tester could watch the display of 
pertinent system data as the simulation operated. When an 
unusual  occurrance was observed, the tester could stop the 
Simulation and query parameters to determine what had taken 
place. Having made his observations, the tester would then 
Mave the choice of restarting the simulation, continuing 
from the point of interrupt, or saving the data produced on 
prd copy. 


9. Limitations 


The simulation that vas programmed had sone 
limitations for any practical use in testing. One of the 
major linitations was size. The number of tasks and other 
elements was limited. This was done to reduce turn-around 
time during the development of the program. Another major 
limitation was the amount of computer time required. The 
Program could have been made to run faster; however, the 
computer time was affected when features were added or 


deleted after most of the simulation had been programmed. 
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